Presentation of the development process of the ALO conceptual model for aligning assessment and Learning Outcomes in Engineering education

1. A Conceptual Model for e-
Assessing Student Learning
in Engineering Education
Rita Falcao
Universidade do Porto :: FEUP
Portugal
ICL 2012 / IGIP 2012 :: Villach, Austria
www.up.pt :: www.fe.up.pt

2. nt of student’s learning and e-learning. It will be focusing in Higher Education
lly in Engineering Education. The question of alignment of e-assessment and ed
Field of study
ed by the area where the three circles overlap.
Learning!
Outcomes!
e0Learning
Assessment! !
Figure 1 - Field of study
www.up.pt :: www.fe.up.pt

3. Initial problem
To what extent e-assessment methods may be
used to measure the achievement of Learning
Outcomes in engineering education?
www.up.pt :: www.fe.up.pt

4. Research questions I
• RQ1) Which type of Learning Outcomes in the
field of Engineering are relevant and should be
considered?
• RQ2) Which are the e-assessment methods
that should be considered?
www.up.pt :: www.fe.up.pt

5. Learning Outcomes
Learning outcomes are statements of what a
learner is expected to know, understand
and/or be able to demonstrate after
completion of learning. (AHELO)
www.up.pt :: www.fe.up.pt

6. Learning Outcomes in EE
Selection and description of LOs in Engineering
Education
• ABET
• EUR-ACE
• CDIO
• EQF
www.up.pt :: www.fe.up.pt

7. (e) Assessment
Assessment: Any procedure used to estimate
student learning for whatever purpose. (Brown et
al)
e-Assessment is the use of ICT and the Internet in
particular for the assessment of learning,
including design, delivery and/or recording of
responses. (JISC)
www.up.pt :: www.fe.up.pt

8. (e) Assessment methods
• Multiple choice questions
• Short Answer Questions
• Problems
• Essays
• Practical work
• Reflexive practice
www.up.pt :: www.fe.up.pt

9. Research questions II
• RQ3) What type of intended Learning
Outcomes can be measured by e- assessment
methods?
• RQ4) Is it possible to propose specific e-
assessment strategies for each type of LO in
EE?
www.up.pt :: www.fe.up.pt

10. The ALIGNMENT question
Evaluation of the Application of e-Learning Methodologies to the Education of
Statement!!of!
the!intented!
Learning!!
Outcome!
Learning!
Outcome!
Teaching!and!
Learning! Assessment!
Tasks!
Ac=vi=es!
Figure 10 - The same LO should be present in all:: www.fe.up.pt
www.up.pt the activities

11. Evaluation of the Application of e-Learning Methodologies to the Education of Engineering
Revised Bloom’s Taxonomy
Table 22 - The Taxonomy Table by Anderson et al.
The Knowledge The Cognitive dimension
dimension
1. Remember 2. Understand 3. Apply 4. Analyze 5. Evaluate 6. Create
A. Factual knowledge
B. Conceptual
knowledge
C. Procedural
knowledge
D. Meta-cognitive
knowledge
The• Taxonomy Table created by Anderson LOs ofplays aqualification frameworks of the
Analyzing and describing the
et al.
the
major role in the development
• Analyzing the describing When approaching the concept
conceptual model inand current work. LOs from case-studies of writing iLOs statements,
several authors refer the importance of including verbs that describe a measurable action so they can
be •
Analyzing and describing assessment methods
assessed [41, 42, 77]. The revised version of the taxonomy of Bloom and the Taxonomy Table
provided the best solution for this problem. The cognitive dimension of the table is structured using
• Aligning assessment and LOs
verbs with detailed descriptions explaining their meaning. Verbs were carefully selected to match
cognitive processes and to be the ones used normally by teachers when:: www.fe.up.pt Additionally,
www.up.pt stating the iLOs.

12. Some pilot tests of application were done using iLO at course level and QF level. It was found that at
the level of QF, most matrixes have a large number of cells filled. At course level, the distribution of
The ALOA conceptual model
filled cells is considerable more precise and easier to match with specific assessment methods. This
represented another revision of the model where alignment was achieved at course level but could be
inferred to the higher levels through existing matrixes.
Figure 15 - Fourth and final version of the model
www.up.pt :: www.fe.up.pt

13. le course usually has more that one iLO statement and more th
ns that one method may be assessing more than one iLO and th
Alignment
than one method. This multiplicity of possibilities is represented i
Many!LO! Many!LOs!
One! Many!
Assessment! assessments!
One!LO! One!LO!
one! Many!
assessment! assessment!
www.up.pt :: www.fe.up.pt
Figure 17 - Alignment possibilities for one unit or course

14. Application of the model
Scenarios Criteria
Verification of internal
Match
alignment
Proposal of internal alignment Emphasis
Vertical alignment Coverage
Horizontal alignment Precision
www.up.pt :: www.fe.up.pt

15. Evaluate: critiquing
Create: planning
Application to EUR-ACE
Table 43 - Complete mapping of EUR-ACE programme outcomes to Bloom's revised taxonomy.
FACTUAL CONCEPTUAL PROCEDURAL METACOGNITIVE
REMEMBER Recognizing 156 156 256
Recalling 156 156 256
UNDERSTAND Interpreting 12456
Exemplifying Evaluation of the Application of e-Learning Methodologies to the Education of Engineering
15
Classifying 15 25
Summarizing 15
Inferring 1245
Comparing 15
Explaining 12456 25
APPLY Executing 2 245
Implementing 2456 23456
ANALYZE Differentiating 456 245
104 Organizing 123456 2345
Attributing 6
EVALUATE Checking 5 5
Critiquing 123456 235
CREATE Generating 2345 35
Planning 23456 2345
Producing
www.up.pt :: www.fe.up.pt

16. Example of case-study
Table 54 - Alignment of LO2 with real assessment
FACTUAL CONCEPTUAL PROCEDURAL METACOGNITIVE
Recognizing CS.02.DSYS.LO.02
Recalling CS.02.DSYS.LO.02
Interpreting
Exemplifying
Classifying
Summarizing
Inferring CS.02.DSYS.LO.02 CS.02.DSYS.LO.02
Comparing CS.02.DSYS.LO.02 CS.02.DSYS.LO.02
Explaining CS.02.DSYS.LO.02 CS.02.DSYS.LO.02
Executing
Implementing
Differentiating
Organizing
Attributing
Checking
Critiquing
Generating
Planning
Producing
www.up.pt :: www.fe.up.pt

17. Thank you!
Rita Falcao de Berredo
ritafalcaoberredo@gmail.com
www.up.pt :: www.fe.up.pt